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Application/Control Number: 09/839,862 Page 2 

Art Unit: 2123 

DETAILED ACTION 

1. This application has been examined. 

2. Claims 1-22 have been examined and rejected. 

Specification 

3. The disclosure is objected to because of the following informalities. Appropriate correction is 
required. 

4. Page 2, summary item 3, "by providing a providing a resuh. . should be corrected to read "by 
providing a result. . 

5. Page 3, last 2 sentences, "a virtual surgery, an image. . ." should read, "a virtual surgery, and an 
image...". 

6. Page 9-10: the description of the elements in Figure 9 do not match up with the drawing. 

Drawings 

7. The drawings are objected to because of the following. Corrected drawing sheets are required in 
reply to the Office action to avoid abandonment of the application. Any amended replacement drawing 
sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only 
one figure is being amended. The figure or figure number of an amended drawing should not be labeled 
as "amended." If a drawing figure is to be canceled, the appropriate figure must be remoyed from the 
replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate 
changes made to the brief description of the several views of the drawings for consistency. Additional 
replacement sheets may be necessary to show the renumbering of the remaining figures. The.repl^ement 
sheet(s) should be labeled "Replacement Sheet" in the page header (as per 37 CFR 1.84(c)) so as not to 
obstruct any portion of the drawing figures. If the changes are not accepted by the examiner, the 
applicant will be notified and informed of any required corrective action in the next Office action. The 
objection to the drawings will not be held in abeyance. 

8. Figures 8 and 9 are objected to because the lines and arrows in the picture overlap the text in the 
drawing, making the text difficult to read. 
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Claim Objections 

9. Claims 3 and 13 are objected to because of the following informalities. Appropriate correction is 
required. 

10. Claims 3 and 15 recite the limitation "taking a photographic picture of the patient's hard tissue". 
"Hard tissue" was interpreted to be the internal structure of the specimen being prepared for surgery such 
as the bones or skeletal structure. Therefore, it is unclear how a photograph will be taken of the hard 
tissue. Further, the specification's explanation of Figure 1 describes that photographic pictures of the 
patient were taken as the picture of the soft tissue and gives no mention to a photograph taken of the hard 
tissue. Therefore, it is unclear as to whether the claim limitation should read "hard tissue" or "soft tissue". 

Claim Rejections - 35 USC§112 

11. The following is a quotation of the first paragraph of 35 U.S.C. 112: 

The specification shall contain a written description of the invention, and of the manner and 
process of making and using it, in such full, clear, concise, and exact terms as to enable any 
person skilled in the art to which it pertains, or with which it is most nearly connected, to make 
and use the same and shall set forth the best mode contemplated by the inventor of carrying out 
his invention. 

12. Claims 10 and 17 are rejected under 35 U.S.C. 112, first paragraph, as failing to comply with the 
enablement requirement. The claim(s) contains subject matter which was not described in the 
specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most 
nearly connected, to make and/or use the invention. 

13. Claims 10 and 17 recite the limitation "the step of generating 2-D pictures of the 3-D 
visualization result", however, the specification (page 5) does not give an adequate description of how 
this step is performed. 

Claim Interpretation 

14. As to the objection to Claims 3 and 13 discussed above, in light of the description of Figure 1 in 
the specification, the claims were interpreted to read "taking a photographic picture of the patient's soft 
tissue", 

15. Claims 10 and 17 recite the limitation "the step of generating 2-D pictures of the 3-D 
visualization result", however, the specification (page 5) does not give an adequate description of how 
this step is performed. The specification states, "2-D pictures from various angles may be generated by 
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projection from the 3-D visualization.. ." (page 5). From this explanation, it was concluded that the 2-D 
pictures generated were just pictures of the model from various angles. 

Claim Rejections - 35 USC §103 

16. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness 
rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as 
set forth in section 102 of this title, if the differences between the subject matter sought to be 
patented and the prior art are such that the subject matter as a whole would have been obvious at 
the time the invention was made to a person having ordinary skill in the art to which said subject 
matter pertains. Patentability shall not be negatived by the manner in which the invention was 
made. 

17. The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), 
that are applied for establishing a background for determining obviousness under 35 U.S.C. 103(a) are 
summarized as follows: 

1 . Determining the scope and contents of the prior art. 

2. Ascertaining the differences between the prior art and the claims at issue. 

3. Resolving the level of ordinary skill in the pertinent art. 

4. Considering objective evidence present in the application indicating obviousness or 
nonobviousness. 

18. Claims 1-22 are rejected under 35 U.S.C. 103(a) as being unpatentable over Forte et al (Forte et 
al, "3D Facial Reconstruction and Visualization of Ancient Egyptian Mummies Using Spiral CT Data", 
ACM SIGGRAPH 99 Conference page 223, 1999), herein referred to as Forte, and further in view of 
Ross et al (U.S. Patent Number 6,608,628), herein referred to as Ross. 

19. As to Claims 1,11 and 18, Forte teaches: taking pictures of the patient's hard and soft tissue 
(Figures 3 and 4, page 5, item 3, page 6, item 5); preparing preprocessing data (page 9, paragraph 1); 
and deriving changes in the patient's soft tissue according to the changes in the hard tissue (page 8, 
second paragraph, "Our aim,..", page 13, last sentence -page 14, first sentence). 

20. Forte teaches that this method will be used for future developments in virtual surgery. However, 
Forte does not expressly teach the performing of virtual surgery by manipulating the patient's hard tissue 
making changes in the hard tissue. 



Application/Control Number: 09/839,862 
Art Unit: 2123 



Page 5 



21 . Ross teaches performing of virtual surgery by manipulating the patient's hard tissue making 
changes in the hard tissue (column 4, lines 23-26,33-42), which includes a reconstruction unit for the 
generation of 3-D images to prepare preprocessing data necessary for virtual surgery (column 4, lines 58- 
6i), wherein the service of a virtual surgery center is used that is connected to a doctor through a network 
(Figure 2, element 22 and description, Figure 12A and description). 

22. Since Forte teaches that the method of deriving changes in the soft tissue as a result of changes in 
hard tissue can be used in virtual surgery, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to use the invention as taught by Forte in connection with the virtual 
surgery system as taught by Ross which includes the manipulation of a patient's hard tissue. 

23. As to Claims 2 and 12, Forte teaches: the method of claim 1, wherein the step of taking data 
comprises the step of taking an x-ray picture of the patient's hard tissue (page 5, section 3, paragraph 2, 
last sentence). 

24. As to Claims 3 and 13, Forte teaches: the method of claim 1, wherein the step of taking data 
comprises the step of taking a photographic picture of the patient's soft tissue (Figure 4, page 6, section 
5, sentence 3). 

25. As to Claims 4 and 14, Forte and Ross teach: the method of claim 1, wherein the step of 
preparing preprocessing data comprises the step of: generating a personalized 3-D model of the patient 
(Forte: page 5, section 3, paragraph 1, Ross: column 4, lines 58-62). 

26. As to Claims 5 and 15, Forte teaches: the method of claim 4, wherein the step of generating a 
personalized 3-D model comprises the steps of: extracting an outline of the patient by overlaying the hard 
tissue picture and the soft tissue picture and extracting feature points of the patient by overlapping the 
outline onto a standard model containing outlines and standard feature points of a representative person 
(page 6, section 5, page 13-page 14). 

27. As to Claims 6-8, Ross teaches: the method of Claim 1 , wherein the step of manipulation 
includes cutting, displacing and rotating the hard tissue (column 4, lines 33-38). 

28. As to Claims 9 and 16, Ross teaches: the method of Claim 1, fijrther comprising the step of 
visualizing the result of the virtual surgery in 3-D (column 11, lines 5-8, lines 22-29). 

29. As to Claims 10 and 17, Forte shows generating 2-D pictures of the 3-D visualization (Figures 
19 and 25) wherein the model is shown from various angles. 

30. As to Claim 19, Ross teaches: the method of claim 1 8, wherein the network is the Internet 
(column 12, lines 31-34). 
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31. As to Claims 20-22, Ross teaches: wherein the virtual surgery center has a virtual surgery 
consulting group for providing consulting service related to virtual surgery (column 11, lines 22-34) 
wherein the group of doctors using this system can include an orthodontic dentist or plastic surgeon. 

Conclusion 

32. The prior art made of record, see PTO 892, and not relied upon is considered pertinent to 
applicant's disclosure, careful consideration must be given prior to Applicant's response to this Office 
Action. 

33. Any inquiry concerning this communication or earlier communications from the examiner should 
be directed to Mary C Hogan whose telephone number is 571-272-3712. The examiner can normally be 
reached on 7:30AM-5PM Monday-Friday. If attempts to reach the examiner by telephone are 
unsuccessful, the examiner's supervisor, Kevin Teska can be reached on 571-272-3716. The fax phone 
number for the organization where this application or proceeding is assigned is 703-872-9306. 
Information regarding the status of an application may be obtained from the Patent Application 
Information Retrieval (PAIR) system. Status information for published applications may be obtained 
from either Private PAIR or Public PAIR. Status information for unpublished applications is available 
through Private PAIR only. For more information about the PAIR system, see http://pair- 
direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 

Mary C Hogan 
Examiner 
Art Unit 2123 



lEO PICARD 
SUPERVISORY PATENT EXAMINER 
TECHNOLOGY CENTER 2100 
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3D facial reconstruction and visualization of ancient Egyptian mummies using spiral CT 

data 

Soft tissues reconstruction and textures application 

Giuseppe Attardi*, Marilina Betro**, Maurizio Forte***, Roberto Gori****, 
Antonella Guidazzoli****, Silvano Imboden*, Francesco Mallegni***** 




* Dipartimento di Informatica, Universita di Pisa 
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• introduction 

• State of the art 

• The research project 

• Project Planning 

• First phase of research: preliminary anthropological results 

• Development of the project: soft tissue reconstruction using VTK 

• Conclusions: the virtual model 
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The problem of rebuilding a face from human remains has been, until now, especially relevant in the ambit of forensic 
sciences, where it is obviously oriented toward the identification of otherwise unrecognizable corpses; but its potential 
interest to archaeologists and anthropologists is not negligible. We present here the preliminary results of a joint 
research among the University of Pisa, the Visualisation Laboratory of CINECA (Bologna) and the CNR-ITABC 
(Institute of Technologies Applied to Cultural Heritage, National Research Council, Rome) whose aim is 
reconstructing, through Spiral Computed Tomography data and virtual modelling techniques (in our case with VTK 
software), 3-D models of the possible physiognomy of ancient egyptian mummies. This work is carried out through a 
multidisciplinary approach, involving different competences: image processing, anthropology, egyptology, computing 
archaeology. 



State of the art 

The application of radiological techniques to Egyptian mummies has a very old and glorious tradition: the first reports 
of a radiological investigation of an Egyptian mummy was published by Petrie in 1 898 [I}, Since then, radiological 
techniques were increasingly used and appreciated throughout the 20* century, as a non-invasive mean of 
investigation: egyptological, anthropological and paleopathological information could be obtained without disturbing 
the mummy?s wrappings. The advent of Computed Tomography in the 1 970?s marked a fiirther milestone in the 
history of mummies? investigation: CT numbers allowed a very fine discrimination between materials with different 
densities, providing an enormous amoimt of information not only about the mummy and its skeleton, but also about the 
artifacts buried wdth the mummy and its coffm [2]. Compared to traditional x-ray techniques, multiple; axial images 
displayed in a clearer way the different details of cartonnage, wrappings, amulets and internal organs of a mummy OJ, 
and allowed easy measurements of exact distances between objects inside or outside the mununy. Since the middle of 
1980?s new developments in computer technology enabled the three-dimensional displaying of axial CT images. The 
new application, bom for clinical use and especially developed for assisting in the plarming of surgical operations, was 
soon extended to munmiies examinations and imaging [41. In the last years, spiral CT has considerabl>>i enhanced 



clinical imaging. The use of this new technique has fiirtherly widened the range and quality of possible investigations 
on egyptian mummies. 

The research project 

The impulse to our research, bom in the ambit of the collaboration between the egyptologists of the University of Pisa 
and the anthropologist Francesco Mallegni, originated in the observation that no previous work dealt with the complex 
problem of repositioning soft tissues on the generated model of the skull. Computerized reconstructions stopped there 
where soft tissues started. Previous works were not specifically interested in the problem of physiognomic 
reconstruction, but, when even the interest existed and plastic models of the mummy?s head were produced, by 
stereolithography or by hand, the final moulding of soft tissues was essentially a "human matter", the joint result of the 
anthropologist?s expertise and the artist?s sensibility £5]. A similar method was already experimented by F. Mallegni 
for the reconstruction of a model of the head of the prince Wadje, whose tomb (about 2000 b.C) was descovered by the 
mission of the University of Pisa, directed by Edda Bresciani £6]. 

The need for an automatic, fast and scientifically based program for the reconstructions of munmiies (and human 
remains) features started the collaboration with the Laboratory of Visualisation of CINECA, mvolved in research both 



Introduction 



http://medialab.di.unipi.it/Project/Mummia/SIGGRAPH99/ 
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on archaeological visualisation and biomedical imaging. 

Focussing on the problem of facial reconstruction, we choosed a mummified head in good condition, from the Egyptian 
Section of the Archaeological Museum in Florence (inv. N. 8643). The date of its acquisition is 1893; we lack any 
other reliable information about its provenance. C14 calibrated dating of a sample of the hair gave a probability 
distribution between 339 b.C. and 201 b.C. [T]. The very good condition of the head, attesting the quality of the 
embalming process, make us prefer the higher dating. 




Fig. 1 The mummified head in Florence (inv. N .8643) (Soprintendenza Archeologica della Toscana) 



Project Planning 

The project involved five different stages: 

L anthropological and egyptological analysis of the head; 

2. spiral CT of the head; 

3. reconstruction of a 3-D model of the skull generated from CT data processing; 

4. reconstruction of soft tissues; 

5. application of textures fitting the somatic features. 

The different stages are not strictly sequential: as we shall see, spiral CT scannings and, later, their 3-D reconstruction 
http://medialab.di.unipi.it/Project/Mununia/SIGGRAPH99/ . 12/6/04 
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provided new interesting data to the previous phases (anthropological and egyptological investigations). 



First phase of research: preliminary anthropological results 

L The anthropological study of the mummified cranial remains allowed us to identify a male subject with an age at 
death of around 40 years. The skull is dolichocranic, of medium height in norma lateralis, and with rounded occiput, 
narrov/ face, high cheekbones, gracile even if v/ell developed in its height, jaw; the orbits are narrow, the nose is well- 
shaped, and of Europoid look. 

The general appearance of the subject, especially regarding the face and the shape and structure of his hair, lead us to 
exclude Negroid influences, but closely resembles present and past Berber ethnic characters. 

The very good conservation of the head pointed to an individual high in the social hierarchy, so as to grant himself an 
effective (and expensive) embalming process. 

2. The mummy was scanned on 18^ April 1997, using a Siemens Somatom Plus 4 spiral computer tomography scanner 
at Careggi Hospital in Florence, thanks to the kind collaboration of the radiological equipe. Slices thickness was 0.5 
mm through all the skull. 



IFig. 2 CT scanning of the head demonstrated the post-mortem transnasal ethmoid 
fracture creaited by the embalmers to extract the brain tissue; the cranial cavity was filled with hot melted resin, 
later solidified, introduced with the mummy resting on its back, as the model reconstructed from the CT images 

clearly displays. 

Embalming excerebration through the ethmoid was very common in the Late Period, practised until Ptolemaic age, as 
well as the filling of the cranial cavity v^th resin {8}. 

The spiral CT images were later electronically transferred to the Onyx2 workstation (Silicon Graphics) at CESfECA for 
post-processing. 

3, In the methodology used for 3-D reconstructions generated by spiral CT data sets, CT slices must be stacked up and 
interpolated in order to build a volume. Once created a volume, it is possible, by means of suitable algorithms, to 
generate surfaces whose points have the same function value. They are called isosurfaces. A popular algorithm for 
determining isosurfaces is the so called marching cubes [9J , the same used in the 3-D reconstruction of our munMny?s 
skull. The principle underlying the application of this algorithm to the kind of problem here described is that sunilar 
materials have the same radio-opacity and are, consequently, represented in a CT scan by the same densitometric level. 
In CT slices, the intensity associated to each pixel in Ae grey-scale is proportional to tissues density: black corresponds 
to air, white to bones. It is therefore possible processing the CT scans sequence so as to obtain a 3-D grid, where to 
each "knot" (control point) is associated the densitometric value measured by the CT scans. The result is a 3-D 256 
grey levels image. 



http://medialab.di.unipi.it/Project/Munmfiia/SIGGRAPH99/ 
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This phase of the work was particularly interesting from the anthropological perspective: the use of this technique 
allowed us to visually exclude the mummified soft tissues and directly observe the cranial bones with a very high 
image resolution. This method offered also the chance of a morphological and morphometric check of the 
anthropologist?s observations on the specimen (covered by the disseccated soft tissue): these two methods of 
investigation led to the same diagnosis of Berber group. 

The image of the skull gave us the chance to observe, directly on the cranial vault, a bone pathology and a biological 
answer to the pathogenic factor that shows a long survival of the subject. X-ray examination of the mummified skull 
showed in more detail the reactions to this kind of pathology so that we could perform a global analysis of the sample. 

4. This stage of our work is still in a preliminary phase. Among the possible methodologies to deal with this complex 
problem, we focussed two different promising ways: 

A, implementation of the anthropologists? protocols developed to the reconstruction of soft tissues on the 
skull; 

use of warping techniques. 

A. Generally, the anthropological methodology to reconstruct soft tissues on a skull is borrowed from forensic 
sciences: as it is well known, the thickness of die soft tissues is reconstructed on the bones through the use of pegs at 
marked points. All the pegs are joined by strips of plastiline of fixed thickness and the empty spaces among them are 
then slowly filled with mouldable material: in this way, it is possible to reconstruct nearly all the face that belonged to 
the living subject; on this, nose cartilage, eye globes and lips are added; because the orbicular muscles around the lips 
leave no impressions on die jaw bones, it is important to consider for their modeling the ethnic group to which the 
subject belonged. The markers helping in the reconstruction are based on anthropology and forensic studies of people 
of varying ages and populations. Their number may vary in the different protocols which anthropologists follow: as for 
the model of Wadje?s head, Francesco Mallegni?s reconstruction was based on the method described by Douglas H. 
Ubelhaker, following Rhine and Campbell?s tables (1980) and Rhine, Mooer and Weston?s (1982) Ufl].. Though this 
method has a certain degree of subjectivity, nevertheless it is sufficiently reliable. A software implementation of the 
above mentioned protocols could assist the operator in locating the markers on the 3-D model of the skull by a graphic 
interface, so as to choose the correct set of anthropological parameters. Subsequently, interpolation methodologies 
could carry out automatically the soft tissues growing. 

B. A different method consists in the distortion (warping) of the 3-D model of a reference scanned head, until its hard 
tissues match those of the mummy. The subsequent stage is die construction of the hybrid model composed by the hard 
tissues of die mummy plus the soft ones of die reference head [11] . 

We believe that very good results could outcome through a semiautomated interactive procedure, integrating the two 
methodologies here described. 

5. While hard and soft tissues give morphological information, textures provide colours and aesthetical features. They 
are "pasted" over die 3D models by means of mapping procedures. In this preliminary phase we used as texture the 
photograph of a modem Berber, published in an andu-opological treatise [12]^ well fitting die general somatic features 
of our reconstruction but, unluckly, of very low resolution (fig. 4). Moreover, being a frontal view, it does not give 
sufficient information for the mapping of the entire model. 
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fig.4 



The texture was mapped onto the 3D model to perfectly match the frontal view of the mummy but it loses its grain as 
soon as we depart from the frontal view. Much better results could be obtained with different high resolution views of a 
new subject. 



I 



Fig.5 The texture, suitably processed and coloured, is mapped onto the 3-D model. 




Fig. 6 Lateral view 
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Devslopinent of the project! soft tissue reconstruction using VTK 

After a first part of work, our open problem is to reconstruct the lacking elements of a 3D digital model generated from 
CT scans applied to a mummified cranial remains (Fig.7) 

As we have described in the previous part of the paper, [14] we work wath an hybrid approach [13]: 




Fig.7 Digital model of the mummy: surfaces corresponding to soft (lef 

on one hand there is the implementation of anthropologists? protocol (also knovm as ?Manchester protocol?) [15] , used 
in manual reconstruction of remain^in order to control the thickness of soft tissues at specific positions in accord to 
the measures indicated in [16] , On the other hand warping techniques allow to enhance the mummy model with 
information coming from another complete CT scarmed head of the same race, with the right properties according to 
anthropological studies, used as reference model. 

Our aim is to obtain a perfect match among hard tissues so that soft tissue of reference model can be used to represent 
those of the mummy with a good approximation. 

Moreover we are developing a tool in order to apply to the model cylindrical textures obtamed multiple views of a well 
suited individual or from other sources as sculptures and paintings. 

Software implementation has been designed using Vtk (The Visualization ToolKit) £17], a public domain library for 
scientific visualization in order to guarantee performance and portability. 

Soft tissues reconstruction 

As shown in fig.8, our methodology may be subdivided in different working steps that we are going to explain deeply. 
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Fig.8 The different steps of soft tissue reconstruction 

For the following steps to work correctly CT scans data representing our model and mummy should have the same 
placing, orientation, dimensions and resolution. This is generally not true especially when dealing with data coming 
from different machine so the first step is to perform a manual registration (Figure 3), that is a rigid transformation, 
among volumes in order to work in the same system of coordinates. Software like AIR [18] are also available for 
automatic registration but sometimes, especially when volumes are quite different, they do not produce satisfactory 
results. , ■ 

As further requirement grey scales of hard tissues must be similar, in spite of different methodologies of acquisition 
though mununy?s tissues has been deteriorated. It is possible to correct these differences shifting and scaling intensities 
using histogram information (Fgs.9-1 0). 



\ 
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Fig.lO Hystogram of mummy and model dataset 
We outline that we prefer to transform the reference model to preserve original data of the mummy. 
For volume resampling, smoothing (to remove aliasing phenomena) and surface generation Vtk internal facilities are 
used. 

We usually take into account isotropic volumes with squared voxel of 1 mm as trade-off between performance and 
image quality. 

At this point we proceed with the setup of the Manchester pegs (see Appendix A) onto the surface of the hard tissues of 
the mummy (fig. 13) while for the reference model it can be predetermined (fig. 14). The aim of these phase is to fix 
some constraints for the resulting physiognomy and to provide a first guess for the follovsdng step that is tht features 
tracking. 

Pegs are mapped onto a spherical surface of parametric ratio, so that the user can place quickly the whole set and the 
adjust single pegs. 
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Fig.l 1 Manchester points placed over the mummy Fig.l2 Bringing the ] 

Calculating vector displacement among couples of corresponding points we obtain a scattered field to drive a first warp 
phase (fig. 11). 
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Fig.l3 Warp driven by Manchester points 
In the next stage we proceed to a warp driven by a features tracking (fig. 13). 

Features tracking consists in determining a correspondence between sets of characteristic points pertaining to the 
volumes in order to obtain a scattered motion field with more details. 

Initially this set of points is chosen as a subset of points that are vertices of hard tissues surface of the mimimy; some of 
these points, with particular characteristics, are identified as features. 

If, consecutively a test, a feature is retained reliable (fig. 14), we search the corresponding position in the reference 
volume. If the result is good, the resulting motion field is defined among subsets of bone surfaces, fi-om the reference 
model to the mummy volume (fig. 15). 

Once generated a scattered motion field, it must be diffused within the whole reference volume. This is performed 
using Shepard method [19] available in Vtk (details are in Appendix B). 

Diffused motion field can be used to warp (one more time using vtk facilities) every structure pertaining to reference 
model coherently with mummy model (figures 16 and 17); therefore we reconstruct mummy soft tissues warping those 
of reference model (fgs. 15-19). 
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Fig.l4 Building blocks for this stage 
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Fig.lS Points on isosurface of bone are used as feature points. Colors represent the Fig,16 An example of a scattered fit 

reliability values 
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Fig.l7 Model skull ( blue) after this stage overlapped with mummy skull (white) 




Fig.] 8 Model skin (blue) and mummy skull (white) 
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Fig.l9 Face generated 

Theory of this algorithm (we will go through a brief explanation) is taken from [20] as a rework of Lukas-Kanade 
algorithm [21]. We have adapted this algorithm to our requirements and built a vtk filter. 

We consider the relation between hard tissues surface of the reference model and hard tissue surface of the mummy as 
a continues deformation in the time. 

If ^ ^ ^ is the intensity of a point of coordinates (x,y,z) at time t in the mummy volume and 

V = (u{x,y^z)Mx,y.z)M^, y,z)) ^ (u,v, w) ^^^^^ ^^^^^ u{x,y,z) ^ vix,y,z) ^ 

^{^yy>^) components in jc,;; e z directions of velocity vector, we suppose that the intensity function is the same 



at the time 



f + « in the point + ^» J' ^ + &) ,f ^^e reference model, where 5x=u5t^dy= vdt 



and^^^O. 

e{x +ua,y +va,z + wa,t+S;) = E{xyy,z,t) 



(1) 

If the intensity function change smoothly with jc,;', z e /, we can manipulate the equation (1) with Taylor?s series to 
obtain 



(2) 



^ ^ ^ cE 
E{x,y,z,t) + + +^ = E{x,y,z,t) 



where e contains terms in 5x, 5y, bz e 5/ higher than first order. 
e(x Z t) 

Eliminating \^>y>^> ) ^ rationing by 5/, and calculating limit for ^ ~^ ^ , we obtain 

+ ^ + + — = 0 

dx dt 8y dt dt dt ^ 
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that is the totally derivative of ^ in the time. 
dE 

— =0 
(4) at 



Using abbreviated notation 
dx 



« = — , 
dt 

E 



dy 
dt 



w = 



dz 

Tt' 



a 



we can write the 3 as 



(5) 



E^u-^-E^v-^E^w-^E^ =0 



known as motion field constraint equation, where E^, Ey E^ ed are partial derivatives. 
We say that x is a reliable feature if 



(6) 

where: 



fVIVl'^dxdydz 

iw(x) 



<> T. 



max 



I(X ^t) is the matrix of intensity function E in the point X =(x,y,z) in the region W(x) at the time t; 
V is the gradient operator; 

a min(4^ ) represents the smaller eigenvalue of matrix ^ ; 
^nun> max are predetermined thresholds. 

We consider a window W (^) centered in q of ^^^^e • WSize • WSiZe dimensions. 
We represent (6) in discrete fashion 



(7) 

where 



max 
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EyE^ E] E^E^ 
E^E, E,E^ E] 



The solution of (4) respect to V is given by 



(8) 
where 

is the 3x3 symmetric matrix that represents the term inside parenthesis of (7), 

A=[V5(Xi) V^(xjf 

and is a timing gradient 

E{x^,t+a)-E{x^yt) 
M 

b= E{Xi,t+&)-E(x^J) 

M 



that is the intensity difference between reference and mummy volumes. 
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Fig.20 In this case the motion field is calculated for a 3x3x3 window and applied in his center (the feature 

point). Bold grid refers to the reference volume. 

Normally this result is affected by an error of 1 0% approximately. To improve its precision we use an iterative 
multistep method: the window is moved in the mummy volume in the direction of estimated motion and the field is 
recalculated among new positions as shown in fig. 21 . 
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Fig.21 At every step the motion field is recalculated between new position. 

The process will be over when displacements become smaller than a predetermined arbitrary constant or when a 
maximum limit to iterations is reached. In case of non convergence the feature is discarded. 
When a feature doesn?t coincide with a voxel, window points are calculated by trilinear interpolation using vtk 
facilities (see fig. 22). 



X 




Fig.22 Intensity value is recalculated by trilinear interpolation of voxels. 

As final stage of soft tissue reconstruction we present warp driven soft tissues. 

In this moment this stage is still in developing so we have no picture, anyway the idea is simple: for each of the 
Manchester points we find its corresponding on the skin surface, in this way we can measure the actual soft tissue 
thickness. By consulting the thickness table we find the corresponding desired thickness measure. 
Saying that the actual thickness must become the desired thickness we generate another scattered field that will drive 
this third warp. 

This last step is performed just on the skin surface. 



Texture application 

We are now at sununarizing mainly points of this stage, still under development, inspired by the work presented in 
1221. 

Starting from at least three views of our candidate, as in figure 23, we will going to generate a cylindrical texture, 
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^^^^^^ 





Fig,23 Several views for generating cylindrical textures 

As shown in figure 24, each point P on the cylindrical texture will be projected on the cylindrical axes, intersecting a 
point q on the surface, watching the angle between the normal in q and the normal of the view we can determine which 
of the views q can see. 

The color for the Point P is given by a weighted sum of the colors taken from the visible views, and the weights depend 
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again oh the angle among the normals. 

Before the extraction from the views colors has to be modified to match the corresponding projection of the whole 
model (fig.25), we can realize this morph implicitly after fixing a way to map projections in the desired points of the 
view. 

This mapping could be achieved by projecting the Manchester points over the views and let an user move them in the 
right place. 




Fig.24 Creating the cylindrical texture 
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Fig.25 Texturized model of recontructed soft tissues of the mummy 



Conclusions: the virtual model 

In our project, for obtaining better performances through the virtual 3D visualisation of the reconstruction we have used 
the powerfiil workstation Onyx2 (Silicon Graphics at VISIT, CINECA), equipped with an architecture of type 
multiprocessor, with 4 processors RIOK, 1 Gbyte of RAM, computing power of 1.5 Gflop, 1 graphic pipeline, that it 
can process 1 1 milions of polygons per second. In fact the main problem, processing a large amount of data, was to 
process and visualise in real time and in 3D the data volxmie. The 

interactive way to explore and show the whole model gives the possibility to understand deep features of the data: in 
our case, our interdisciplinary team, constituted by archaeologists, anthropologists, and computer experts, has could 
discuss in real time problems and characteristics of the data comparing in detail hypotesis and interpretations about the 
egyptian head, anthropometric data and computer/images information. We can describe the processing as a cognitive 
model of the find. Finally, we have experimented the VR CrystalEyes, wireless pairs of glasses, v/ith lenses capable of 
alternately shuttering in sequence with left-eye right-eye images 

interlaced on a computer display; the result is a stereoscopic effect, allowing the user to view the contents of the 
computer display in 3D. 

Possible develpoments of the project will concern: 

• Integration of CT scan data and reconstructions with the egyptian iconography (i.e. ancient sculptures of the 
same period). 

• VRML version of that kind of models. 

• Images databases on line in Internet for organising a world collection of this type of data. 

• Project integration with other sciences with the same aims, such as forensic sciences, virtual surgery, biology, 
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criminology. 

• Off line multimedia version of data (i.e. on CD ROM or DVD), for publications and scientific divulgation. 

Finally, although some modules are still in development and reference model was not ideal for this case of study 
because it is an European, we think to have obtained satisfactory results, that is an Egyptian physiognomy with some 
European element. 

The level of automation reached in building models from CT data, reconstruction, texture application and visualization 
allow to the user to complete whole process in 2-3 hours on a PC or graphic workstation . Moreover we hope to reduce 
time consuniing phases like features tracking, that could be improved experimenting others algorithms of non-rigid 
registration. 



Appendix A: our set of skull markers 

We have referred to the set of points illustrated onto tables by Rhine & Moore . These points, originally is in number of 
32 and mainly concentrated on the face, are in green in figure 26. 

For our purpose we add further points, the yellow ones (figure 27 ) and now we have a set of 67. 




Fig.26 Original points for Manchester protocol 
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Fig.27 Extended set of points 



Appendix B: detail for diffusing scattered fields and warp 

Our building block to perform diffusion is the Shepard?s method that is implemented by the vtkShepardMethod filter 
oftheVtk library. 

Shepard?s method is an example of a basis function method , more specifically is an inverse distande weighted 
interpolation technique that can be written as: 




To make this filter operate on vector values we write a new Filter that internally instantiate three vtkShepardMethod 
and let them operate on the single components as show in next figure; 
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PAY RAISE.. .Russian president Boris Yeltsin has approved substandial salary increases for top scientists in an effort to stop the 
brain drain of scientific talent out of his country and strengthen political support within the scientific community. Implemented last 
month, full members of the Russian Academy of Science receive monthly stipends of 150,000 rubles (approx. $128) on top of their 
regular salaries. Corresponding members of the academy receive a stipend of half this value. There will also be increases for 
qualified researchers and for postgraduate students. The regret in the scientific community is that Yeltsin's moves will not be enough 
to help impoverished Russian scientists, much less abandon their idea of emigration. 

TV PRINT OUT... Hewlett Packard has developed a device resembling a thin VCR that will allow users to link a color printer to their 
television. HP is teaming with Time Wamer next April in an experiment based in Orlando Fla., where residents will have wide access 
to interactive TV services. Participants will be able to print coupons, shopping brochures, or articles retrieved via TVs. The device 
works only with HP printers and doesn't yet offer photolike quality. People taking part in the Oriando project will be able to use the 
device without charge; the martlet price will be about $3,500. HP says the home version must sell for $300 to be a hit. 

FLICKERING OPPOSITION. ..Stabilizing the flicker of computer screens could have the opposite effect on eye strain, according to 
psychologists. Nature reports screen flickering occurs each time a cathode ray tube refreshes the screen with new infomnation, 
approximately 50-60 times a second. Increasing this rate to 100 times a second has been suggested to stabilize the screen, 
therefore alleviating eye strain. The controlled study of how screen flicker directly affects the minute unconscious eye movements 
used in screen reading suggests eye movements are increasingly dismpted when screen refi^eshing is raised to the suggested rate. 

COMPUTER STATS.. .Of the more than 90% of the people who say they use a computer at work, Priority Management reports that 
18% describe themselves as very computer skilled. 17% say they have basic skills, 29% report they are adequate and 36% say they 
are inadequate. 

LABORATORY EXPERIMENT.. .Due to financial and organizational problems, an attempt to establish a Japanese version of the 
M.I.T. Media Lab faces an uncertain future. The project was started in 1987, its goal being to build a research facility outside Tokyo 
with the creative agenda and style traits associated with the M.I.T. lab. However, Japan's current economic stump has hindered the 
ability of industries to spend money on advanced research. There has also been reports of a clash over organizational strategy 
between some Japanese organizers and Nicholas Negroponte, the director of the Media Lab. M.l.T.'s five-year contract with the 
International Media Research Foundation expired in June 1992 and since then six people have stepped down from the board of 
directors, including the chair. Tadashi Sasaki, one of Japan's most prominent engineers. The secretary general of the foundation 
said it still plans to build the laboratory by 1996. 



http://proquest.umi.com/pqdweb?index=0&sid=2&srchmode=l&vinst=PROD&fmt^^ 12/6/04 



Document View 



Page 2 of 2 



-SEARCH TEAM.. .Veterans can search for old military friends by using a new computerized locator service. VET-FIND Inc.. based in 
Las Vegas, is unique in its ability to Identify someone using only a few key pieces of information. For a one-time fee of $35.00 
veterans are pemianently enrolled In the database and may search for as many as four friends. Unfulfilled searches contmue 
indefinitely and privacy is assured. For more information call 800-838-7787. 

E-PORN ARRESTS. ..Six people have been charged in an ongoing effort by federal law enforcement officials to crack an 
international pornography ring that has flourished using an electronic bulletin board. Computenvorld reports the Denmark-basecJ 
board, known as Bamse. can be used to download a variety of child-related pornography in the form of graphic images, text, and 
computer games. More charges are pending as of this writing. 

PUT A MUZZLE ON IT.. .An antinoise generator, a device designed to reduce the sound of buses, is being tested in New York. 
Houston and two transit systems in Southern California. A computer-driven 220-watt speaker, invented by Noise Cancellation 
Technologies Inc.. Stamford, Conn., is placed in the tailpipe of the vehicle and emits sound waves to partially cancel exhaust noise. 
But the New York Times reports the electronic muffler is of greater interest to transit officials for its fuel efficiency traits: the 
technology makes it easier for the bus to emit exhaust which in turn raises horsepower and increases fuel economy. Further muffling 
technology may eventually be able to quiet transmissions, drivetrains and fans or air-conditioning compressors, 

ALL THE RIGHT MOVES...A New York dentist has invented a virtual surgery system that allows oral surgeons to practice delicate 
operations as many times as necessary. Digitalized data on the patient's teeth, gums and tissues are entered into the system using 
a wand-like device passed over the surgical site. X-ray findings are incorporated into this mouth map on which the dentist can 
rehearse. 

SORRY, CHARLIE.. .In a move to protect their property, Avis. Hertz and other car rental companies are using electronic links to 
government computers to screen potential renters. The electronic checking, which only takes seconds, is turning away 6% to 10% of 
the people who request cars. Clerks enter the potential renter's name, birth date and driver's license number on the rental 
company's central computer and it is routed to a company in Queens, NY, which then pulls a public record at the motor vehicle 
department of the appropriate state. The record is analyzed according to criteria set by the rental company. 

SPATIAL CONSIDERATIONS..,Finding a legal parking space in New York City requires quick and cunning moves. However, City 
College engineering supervisor Randolph Archbald went a little too far when he started turning out bogus parking permits using the 
college's computers and his own sophisticated equipment. He was soon nabbed and charged with counterfeiting police and 
correction department parking placards. Authorities said Archbald used an expensive laser wand that made it simple to duplicate 
genuine permits which would allow him to park anywhere (except in front of hydrants and bus stops). Archbald is the first person 
apprehended for producing fake permits, if convicted, he faces a maximum jail term of 15 years. 
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Abstract: Surgical training is one of the most promising areas in medicine where 3-D computer 
graphics and virtual reality techniques are emerging. This paper provides an advanced report on 
our project that aims to create virtual reality tools for training medical students and for 
improving skills and efficiency of orthopedic surgeons in internal fixation of bone fractures. The 
paper describes the whole pipeline from the CT scanner through the Unix-workstation to the 
personal computer where eventually the program works. We also describe the methods and 
mathematical models that allowed us to implement the ultimate goal of the project — to allow the 
surgeons to perform training surgical operations even at their home computers without any 
special expensive hardware devices. This project was initiated in co-operation between NTU and 
SGH and currently continues as an international project. 

Keywords: surgery training, virtual reality, orthopedics. ' 



L Introduction 

Surgical training is one of the most promising areas in medicine where 3-D computer graphics 
and virtual reality techniques are emerging. Attention of researchers has now turned toward 
using combined 3-D reconstruction and virtual environment (VE) technologies to train 
clinicians and to help surgeons plan patient-specific, complex, procedures in craniofacial 
reconstruction and plastic surgery, surgery for trauma fi-om accidents, breast cancer detection, 
and reconstruction surgery. 1-6 In surgery simulation for training the clinicians, the benefit of 
savings in time, cost, equipment, and safety can be enormous, As a result, there is a growing 
interest in applying VR simulation technology to the medical educational process. 

Surgical simulators are used to create a VE wherein the surgeon may simulate and plan die 
operation, and the surgeons or students can be trained. One of the main problems is to receive a 
photo-realistic representation of operation room, human body parts from CT or MRI data, 
surgical instruments and implants. Sound can be imposed into the scene to add realism. For 
example, sound of drilling has been added in VE of craniofacial surgery simulation. 1 Virtual 
surgery is often performed with tissue and blood.5-8 For example, the training system 
KJSMET7,8 provides simulation techniques, which allow the modeling of "virtual tissue", 
based on a data-model which reflects the physical characteristics like mass, stiffness and 
damping of real tissue. However, for orthopedic surgery, it is not necessary to simulate the soft 
tissue except of some cases in craniofacial surgery. For example, it is important to know the 
result of bones replacement covered with tissue in craniofacial sxirgery. When all the 
reconstruction work is completed, the soft tissues are replaced so that die surgeon can visualize 
the result. If the outcome of die virtual surgery is not desirable to the surgeon, the virtual 
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process can be repeated until satisfaction is obtained, without touching the patient. In surgical 
simulators, a collision test algorithm for detection of the contact between surgical instruments 
and the virtual organs is often performed. In the training system KISMET , contact forces 
between the tissue and the instrument end-effector are calculated which can be used to drive a 
force-reflecting surgeon interface 3D-simulation support during the design of new surgical 
instruments and manipulator. 

A comprehensive software system for interactive visualization, manipulation and 
measurement of multi-modality 3-D medical images has been developed, used and evaluated at 
the Mayo Clinic for Computer Aided Surgery and Radiation Treatment Planning for more than 
a decade.''*** This software system, called ANALYZE, has provided surgeons and physicians 
with powerful and flexible computational support both for pre-operative surgical and treatment 
planning and for post-operative evaluation. The software has been applied to a variety of 
surgical and medical problems, and used on significant numbers of patients at the Mayo Clinic 
and at many other institutions. 

The training systems available on the market are oriented on solving the particular 
problems. For example, Limb Trauma Simulator (Musculographics Inc.) models a gunshot 
wound to the leg for military surgeons training.^' Software for VR Surgery mainly works on 
UNIX on Silicon workstations or is prepared on the powerftil workstations. For example, 
Voxel-Man Junior is a subproject of the general VOXEL-MAN-project aiming to transfer 
substantial functions of exploring and manipulating 3D anatomical models from the workstation 
based VOXEL-MAN system to inexpensive PCs.*^'" 

In this paper, we address the problem of developing virtual reality tools for training medical 
students and for improving skills and efficiency of orthopedic surgeons in internal fixation of 
bone fractures. We developed a desktop VR orthopedic surgery training system capable of 
running on commonly available PCs including notebooks without compulsory use of VR 
hardware. 



2. The Problem Statement 

During common orthopedic surgery training, students must fix fractures on synthetic plastic 
bones using surgical tools and implants. Fig. 1 shows a typical fractured plastic bone. These 
synthetic bones differ in quality and cost. Good-quality synthetic bones aren't just simple plastic 
dummies. They are made from the materials with different densities and textures on the surface 
and inside to simulate real bones as closely as possible. ' 



Normally, students assemble the bone in front of them, in the position typical of thfc 
respective real operation. Then, they reduce the fracture and fix it internally using implants that 
include hundreds of different plates, screws, nails, and wires. See Fig. 2, for example, where the 
fractured femur is fixed with the nail inserted in its canal and with four screws. For the insertion 
of implants, students — or surgeons — use different surgical instruments that let them drill holes, 
measure for a length, insert the implants, and so forth. The training lab with all this stuff looks 
rather like a repair shop. There is no flesh and blood there, or anesthesia and assisting nurses. 
Each student works along independently. 
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Fig. 1 . Fractured synthetic bone. 





Fig. 2. X-ray of the fracture and its fix. 



The objectives of this orthopedic training follow: 

• To memorize the techniques and respective tools and implants used for fixing fractures, and 
to choose the most relevant technique for each case. 

• To learn how to put the implants in place, including exactly where and how deep to drill or 
cut the bone, the proper insertion angle, and so on. 

• To acquire "muscle memory " for the physical efforts applied to the tools and implants to 
avoid damaging the bones. 

Many synthetic bones are spoilt before a student can demonstrate the necessary skills on 
this "synthetic patient." The next step is surgical operations on cadavers. Only after that is the 
student allowed to approach a real patient. 

The idea of using a computer for orthopedic surgery training came from surgeons in the 
Department of Orthopedics of Singapore General Hospital, who approached the School of 
Applied Science at Nanyang Technological University (where the first two authors were 
employed at the time). The surgeons were interested, first, in saving the cost of the bones — 
good-quality synthetic bones are expensive — and second, in the ability to work with certain 
types of bones for which synthetic models might not be available. The latter need also derived 
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from the fact that real Asian bones have their own geometric specifics compared to the 
commonly available synthetic bones. Besides that, the surgeons anticipated an attractive 
possibility — training the students on virtual models of real fractures obtained from computed 
tomography (CT) or magnetic resonance imaging (MRI) data. 

Our feasibility study assured us that at least the first two objectives of the common 
orthopedic training could move into the virtual environment relatively easily, with the outside 
possibility of complete virtual orthopedic surgery training if special haptic devices were used. 
Certainly, this virtual training won't substitute completely for the compulsory training on 
synthetic bones and on cadavers, but it might let the students perform the initial routine work 
entirely in the virtual environment, thus saving cost and time. 

To make this virtual training easily accessible, we decided to develop a software system 
capable of running on PCs, available in every medical clinic and in homes. This desktop or 
"pocket" VR system had to produce acceptable realism even without special VR input devices 
such as head-mounted displays (HMDs) or goggles, although using them obviously would 
provide better immersion. 

Eventually, we came up with a project aiming to develop an inexpensive alternative to 
common orthopedic surgery training that would allow the surgeons the following: 

• To keep in the computer every single bone of different sizes and specific features that might 
be difficult and expensive to obtain using synthetic models. 

• To create models of the fractured bones from real patients using data from CT and MRI 
images. 

• To perform surgical operation planning before undertaking the actual operation. 
3. Virtual Orthopedic Surgery Training 

First initial reports on this project were published in papers.'^ '^ In this paper, we provide an 
extended report on the project describing the whole pipeline from the CT scanner through the 
SGI-workstation to the personal computer where eventually the program works. We also 
describe the methods and mathematical models that we used for implementing the ultimate goal 
of the project — to allow surgeons to perform virtual surgical operations even at their home 
computers without any special expensive hardware devices. 

Achieving inrunersion in most VR systems usually requires the following: 

1 . Realistic 3D geometric models with behavior and constraints. 

2. Real-time simulation including collision detection, sounds, and so forth. 

3. Real-time 3D rendering. 

4. VR rendering and input techniques based on special hardware devices. 

Since we aimed to develop a system capable of running on PCs including notebooks 
without compulsory use of VR hardware, we concentrated on the first three requirements at the 
current stage of the project. Also, while designing the system, we kept in mind the significant 
difference between computer-assisted surgery and training systems. The former assist and guide 
the surgeon during the real surgery operation, while the latter simulate only certain aspects of 
these operations with a sufficient degree of realism. 

In the virtual environment, the orthopedic surgeon deals with models of bones, surgical 
tools, and implants. Fixing fractures requires using certain virtual fixation techniques. To 
achieve our goals, we had to find a reasonable balance among the processing time, the size, and 
the complexity of the models and techniques. 
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4. The Data Pipeline 

Except for cases where the data for reconstruction of the bones, the tools, and the implants come 
from the atlases or third parties, the following pipeline was developed for the project. 

Initially, the CT-images are obtained. Images like those depicted in Fig. 3 are to be 
delivered to a computer that is responsible for 3D model reconstruction. Normally, these images 
come in the proprietary data formats and have to be converted in one of the conunon image 
formats, hi our case, for all these purposes, we used SGI workstations though it could be 
virtually any Unix workstation or Linux computer. The choice of SGI workstation was done 
simply because we planned to use the 3D reconstruction software that was available on it. 
Actually, the same processes could be implemented entirely on the personal computer. 

After the CT images are copied to the workstation, there are basically two ways to work 
with them. 

First, they can be automatically processed by the special software that reconstructs 3D 
models from the 2D slices. For these purposes, we used the free-ware software NUAGES by 
Bemhard Geiger'^ that creates 3D polygonal meshes from the 2D contours. 

Fig. 3. Example of CT-images. Fig. 4. A 3D reconstruction from the images. 
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Fig. 5. The interactive 3D modeling tool for extracting and editing contour lines. 

Second, for the editing purposes, we developed an interactive 3D tool that builds and edits contour 
lines for the CT slices and converts them to the data file later processed by NUAGES, In Fig. 4, an 
example of the 3D reconstruction from the data in Fig. 3 is depicted. In Figs, 5 and 6, the 3D editing tool, 
and the reconstructed models of the femur are presented. The 3D data of the bones obtained later are 
converted to the data format used by our program and then transferred to the personal computer. 



Fig. 6. The reconstructed femur and the femur neck 



5. Objects and Operations in the VE 

Analyzing the application area, we understood that most bone fractures appear to occur in 
predictable patterns. For example, there are 27 types of femoral fractures (see Fig. 7) and 9 
types of pelvic fractures. This encouraged us to create a geometric database of the fractured 
bones using geometric models of the broken bones created from standardized geometric models. 
These bones were "broken" as needed using interactive modeling tools and other methods. 

To fix fractures, surgeons use hundreds of different screws, plates, nails, wires, and locking 
bolts. Their geometric models and the models of the tools were also created and stored in the 
respective geometric databases (see Fig. 8). 
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Since the software had to be implemented on a common personal computer, we had to find 
a reasonable balance between the processing time, the size, and the complexity of the model. 
Analyzing the goals that must be achieved, probably the straightforward method can be to 
implement something like the ftinctionally based spatial spline.*' This spline would interpolate 
the bones from the scattered points, and for any trial point would answer the question whether 
the point is inside, outside or on the boundary of the bone. Unfortunately, all these methods, 
providing a genera! solution to practically every problem, assxmie very extensive use of 
computer memory, and will not provide the required interactivity. Thus, we decided to combine 
different models for different purposes in our representations. 




Fig. 7. Several femoral firactures from the geometric database of common fractures 

implemented for the project. 




Fig. 8. Real and virtual surgical tools and implants from the geometric database implemented 

for the project. 

Internally, each geometric model in the VE is represented as a polygonal mesh with the 
functional description that — like geometric DNA — constitutes its shape. Extra application data 
is stored in the attachment to the geometric model. Thus, for example, extra information about 
the number, the size, and the locations of the holes for each plate are stored with the polygonal 
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mesh and functional description of the plate. For the bones, the essential infonnation is the 
location of their medial axes. 



Since the final assembly somewhat resembles complex objects in CAD, each object in the \ 
VE and the scenes composed from the objects are stored in the hierarchical geometric database. 
For example, each plate with holes is a hierarchy of the solid plate body as a root object with the 
addressable holes as its children nodes. When inserting a screw through the plate, first the 
surgeon picks up the hole, then the inserted screw becomes yet another child node of this 
assembly. If the guide wire is inserted through the plate's hole, later it will be used for inserting 
the cannulated screw. In that case, the coordinate systems of the guide wire and the cannulated 
screw will be aligned to provide the guiding effect. After the insertion, the guide wire will be 
removed from the scene, and the screw will substitute for it in the hierarchy. 

Many common surgical techniques handle internal fixation of bone fractures, such as 
fixation with cancellous screws^**, fixation with an intra-meduUary nail^*, dynamic condylar 
screw (DCS) implant system technique^^ dynamic hip screw (DHS) implant system technique^\ 
and so on. Analyzing them systematically from a geometric point of view, we managed to find 
common and specific parts that let us come up with the generic virtual fixation techniques that 
became foundation virtual methods in our project. These methods have been implemented on 
the internal level, while the surgeon sees the familiar names and commands at the user interface. 



6. Collision Detection 

All the geometric and nongeometric data are used together for pseudo-physical collision 
detection — crucial for VR modeling. Each particular operation required developing its own 
respective methods and models. 

For example, when inserting a screw into bone through a plate's hole, the proximity sensors 
of each plate implemented as fimctional fields engage to attract the screw tip to the nearest hole. 
Then, depending on the mode selected, the screw will either automatically align with the hole 
axis or the surgeon will have a limited ability to change its orientation, thus simulating the 
actual insertion of a screw. The screw being inserted never goes through the solid parts of the 
plate. Since the length of each screw and all the sizes of the plate are among the parameters 
stored in the geometric database, the screw never penetrates deeper than it should. For the 
surgeon, the result looks very realistic. 

Another example of handling collision detection involves an implementation of na^l 
insertion in the femoral canal (Fig. 9a). The femur with its canal has a curvature of a certain 
radius. The nail to be inserted (Fig. 9b) is a piece of a large torus with the same curvature as that 
of the femur. 

Mathematically, the problem of insertion is to find the initial location of the torus so that it 
lies in the same plane with the bone's medial axis and so that its axis is coincident with the axis 
of the bone curvature. From the surgeon's point of view, the skill required in this operation is to 
find the proper point and angle of insertion for the nail. These initial parameters let us solve the 
geometric problem of the axes' coincidence and answer the question whether in principle the 
nail goes freely all the way through the canal or seriously damages the bone. 

These are just two examples of the methods used to solve collision detection problems in 
our project. Each particular operation required developing its own respective methods and 
techniques. 
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Fig. 9. Universal femoral nail. 



7. Virtual Bone-setter 

We called the software implemented in this project Virtual Bone-setter. The program runs on 
PCs with the most common architecture though the following features are preferred: 

• Powerful CPU not less than 1 50 MHz. 

• Extended RAM 64MB or above. 

• Video card supporting 3D polygon rendering and real transparency. 

Using a mouse as an input device controlling object relocations may suffice, but for some 
users it might not be as realistic as expected. Better immersion results from using relatively 
cheap 3D mice or graphics pads instead. The software is implemented with Criterion's 
Renderware, which provides the required interactivity with a reasonable quality of rendering. 
Comparing with the de facto standard OpenGL, it provides better speed of rendering though 
slightly sacrificing the rendering quality since it uses less number of bits for representing the 
colors. The ability of Renderware to model in hierarchical coordinate systems and its support of 
many VR devices provides a very natural environment for the geometric modeling in the 
project. 

Starting the virtual surgery, the surgeon locates the fractured bone in 3D space looking at 
the screen. Then, depending on the fracture, the surgeon can issue the following basic 
commands: 
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• Applying the instruments and putting the implants in place. 

To simulate the surgical techniques, the following explicit and implicit commands are used: 
"Insert threaded guide wire", "Insert pin", "Remove threaded guide wire", "Remove pin", 
"Insert multiple guide wires", "Measure for the screw length". "Insert screw", "Seat plate", 
"Insert nail", etc. These operations are applied to the current objects in the scene. The user 
is required to locate the application point on the bone, to define the orientation or to follow 
the guided orientation, and to apply or to insert the instrument or implant to the proper 
depth. In common training where expensive non-reusable synthetic bones are operated, it 
usually takes hours of practicing to achieve the required skills. All the models were created 
with the maximum possible realism to train the siu*geons visually so that they will 
remember how all this hardware looks before they enter the training lab. Collision detection 
is implemented to ensure proper insertion of the implants (e.g., for the screws being inserted 
through the plate holes and for the cannulated screws being inserted by the guiding wires). 
The screws rotate about their axes when inserted. The speed of insertion slows down when 
the tension is growing. Drilling and cracking sounds are played for the realistic audio 
' feedback. 

• Movin^> rotating, and zooming the scene and the objects in the scene . 

The whole scene or individual objects can be moved, rotated or zoomed in any direction. 

• Looking at the assembly through the X-rav lens. 

The surgeon often inserts the wires, nails or pins to the appropriate depth under image 
intensification pC-ray). In the virtual environment, it is provided either for the whole scene 
or for the particular part of it through the so-called X-ray lens. 

• Walking throu|gh the bone canal. 

It allows the surgeon to look at the seated implants from inside the bone. This effect can not 
be achieved during the real surgery and is implemented here for education and control 
purposes. 

• ' Reversing process. 

If something has been done wrong, the multilevel undo operation can be applied without 
any damage to the models. 

• Setting the lights and the background. 

Different type of lights can be applied for better illumination of the surgical field thus 
simulating the actual environment in the operation theatre. These can be onmi-directional or 
spot-lights. 

The program guides the surgeon through the whole surgery, although the surgeon retains the 
ability to make independent decisions. Pseudo-physical collision detection is implemented. 
Realistic sounds play when the instruments are used. The scene with the fi^cture being fixed can 
be saved at any time and stored in the hierarchical geometric database for fiirther use. If the 
surgeon requires an implant not in the database, its model can be created interactively and used 
immediately. If the surgeon wishes to study a real fracture rather than those stored in the 
fracture database, its geometric model can be reconstructed from CT data. 

8. Examples of Virtual Fracture Fixation 

Femoral neck fracture fixation with cancellous screws, as depicted in Fig. 10,requires the 
following actions: 
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Fig. 10. Virtual femoral neck fracture fixation with cancellous screws. 



1 . The first step includes guide wire insertion to the appropriate depth. The surgeon chooses 
"Insert threaded guide wire "and locates the point of insertion on the bone. The wire appears 
on the screen touching the bone at the selected point. The surgeon rotates the wire to the 
proper insertion angle and inserts it. The result of the insertion can be checked with the 
simulated X-ray imaging. 

2. The next step is placing multiple parallel guide wires at various distances from the first 
wire. The surgeon selects "Place multiple guide wires "and defines the respective points on 
the bone. Wires are inserted automatically parallel to the first one, thus implementing the 
adjustable parallel wire guide device. 

3. Then, the surgeon measures for the screw length. The reading indicates the appropriate 
screw length. It simulates the result of applying the cannulated screw-measuring device. 

4. Next comes inserting the screws. The surgeon selects the cannulated screw of the 
appropriate length from the database, picks up the wire, and places the screw over the wire. 
After that, other cannulated screws are placed over the respective wires. 

5. The last step is to remove and discard the guide wires. 

Another example is the inter-trochanteric fracture fixed with a DHS implant system (see 
Fig. 11). In this case; the surgeon inserts the guide wire first, next the guide pin, and then 
removes the wire. The surgeon measures the pin for a screw length and inserts the DHS screw 
with the appropriate length over the pin. After that, the surgeon seats the appropriate DHS plate 
over the screw, removes the pin, and fixes the plate with cortex screws through the plate holes. 
Finally, the surgeon applies the DHS compressing screw to secure the assembly. 
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Fig. 1 1 . Virtual femoral neck fracture fixation with DHS implant system. 

Yet another example of the virtual femoral fracture fixation with an intra-meduUary nail 
appears in Fig. 12, This VR surgery simulates the real surgery illustrated by the x-rays in 
Fig. 2 and by the diagram in Fig. 9a. Here, the nail— in fact, a piece of a torus — is to be inserted 
in the bone canal and then fixed with the screws. This operation requires hours of training 
before the surgeon learns exactly where to insert the nail and what the insertion angle should be. 



Fig. 12. Virtual femoral fracture fixation with an intra-medullary nail. 



9. Conclusion 

The implemented tools let surgeons learn how to fix fractured bones and perfomi preoperative 
planning without wasting expensive synthetic bones. The software works on common PCs and 
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simulates surgical techniques, implants, and tools. Using more sophisticated virtual input/output 
devices, although not compulsory, increases realism and provides better unmersion. Besides 
that, our software also may be used for simulating internal operations for some bone diseases, 
like slipped capital femoral epiphyses, ankle arthrodeses, sacroiliac joint disruptions, subtalar 
arthrodeses, and so forth. 

The long-term goals of the project follow: 

• Photorealistic representation of the operating room environment. 

• Force feedback and tactile response using VR haptic devices. 

• Object-management at different levels-of-detail. 

• Greater inunersion, letting the surgeon study and manipulate virtual objects on the screen or 
on a virtual workbench with a higher degree of realism. 

These goals assume software development in both directions — on PCs and advanced 
software implemented on a virtual workbench. Both lines of software will use the same models 
and data format, allowing surgeons to handle simple cases on their home or office computers 
and to ring the most important or complex cases to expensive graphics workstations. 
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NEW CENTRAL FAX NUMBER 

Effective July 15, 2005 



On July 15.2005 . the Central FAX Number will change to 571-273-8300. 
This new Central FAX Number is the result of relocating the Central FAX 
server to the Office's Alexandria, Virginia campus. 

Most facsimile-transmitted patent application related correspondence is 
required to be sent to the Central FAX Number. To give customers time to 
adjust to the new Central FAX Number, faxes sent to the old number (703- 
872-9306) will be routed to the new number until September 15, 2005. 
After September 15. 2005 , the old number will no longer be in service and 
571-273-8300 will be the only facsimile number recognized for "centralized 
delivery". 

CENTRALIZED DELIVERY POLICY: For patent related correspondence, 
hand carry deliveries must be made to the Customer Service Window (now 
located at the Randolph Building, 401 Dulany Street, Alexandria, VA 
223 14), and facsimile transmissions must be sent to the Central FAX 
number, unless an exception applies. For example, if the examiner has 
rejected claims in a regular U.S. patent application, and the reply to the 
examiner's Office action is desired to be transmitted by facsimile rather than 
mailed, the reply must be sent to the Central FAX Number. 
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